Capstone Project

Pneumonia Detection

Problem Statement: Chest Radiograph is the most commonly used or performed diagnostic imaging Technology. Due to high volume of chest radiography, it could be very time consuming and intensive for the radiologists to review each image manually. As such, an automated solution is ideal to locate the position of inflammation in an image. By having such an automated pneumonia screening system, this can assist physicians to make better clinical decisions.

Business Domain Value: Automating Pneumonia screening in chest radiographs, providing affected area details through bounding box. Assist physicians to make better clinical decisions or even replace human judgement in certain functional areas of healthcare (eg, radiology). Guided by relevant clinical questions, powerful AI techniques can unlock clinically relevant information hidden in the massive amount of data, which in turn can assist clinical decision making.

Details about the data and dataset files are given in below link, https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/data

Purpose of this project: Find out a patient with pnuemonia disease. There are some patients, which have some symptoms but not sure they might get affected by this diesea or not. This project also help to find out those patients, who have some symptoms and might get affected by this disease.

from google.colab import drive
drive.mount('/content/drive')

Mounted at /content/drive

cd '/content/drive/MyDrive/Colab Notebooks/Capstone Project'

/content/drive/MyDrive/Colab Notebooks/Capstone Project

!ls

'Capstone Project Pneumonia Detection Ronald.ipynb'
'GCP Credits Request Link - RSNA.txt'
 Pneumonia_Detection.ipynb
'Pneumonia Detection Ronald.ipynb'
'Pneumonia Detection with PPTs'
 stage_2_detailed_class_info.csv
 stage_2_sample_submission.csv
 stage_2_train_labels.csv

!pip3 install pydicom
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import pydicom as dcm
import os
import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
import cv2
import gc
import warnings
warnings.filterwarnings("ignore")

Collecting pydicom
  Downloading pydicom-2.2.2-py3-none-any.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 5.3 MB/s 
Installing collected packages: pydicom
Successfully installed pydicom-2.2.2

#Unzip the .zip file

#from zipfile import ZipFile

#with ZipFile('/content/drive/MyDrive/Colab Notebooks/Capstone Project/rsna-pneumonia-detection-challenge.zip', 'r') as z:
  #z.extractall()

def read_csv(file_path):
  content = pd.read_csv(file_path)
  return content

project_path = '/content/drive/MyDrive/Colab Notebooks/Capstone Project/'
labels = pd.read_csv("/".join((project_path, 'stage_2_train_labels.csv')))
class_desc = pd.read_csv("/".join((project_path, 'stage_2_detailed_class_info.csv')))

labels.head()

	patientId	x	y	width	height	Target
0	0004cfab-14fd-4e49-80ba-63a80b6bddd6	NaN	NaN	NaN	NaN	0
1	00313ee0-9eaa-42f4-b0ab-c148ed3241cd	NaN	NaN	NaN	NaN	0
2	00322d4d-1c29-4943-afc9-b6754be640eb	NaN	NaN	NaN	NaN	0
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	NaN	NaN	NaN	NaN	0
4	00436515-870c-4b36-a041-de91049b9ab4	264.0	152.0	213.0	379.0	1

class_desc.head()

	patientId	class
0	0004cfab-14fd-4e49-80ba-63a80b6bddd6	No Lung Opacity / Not Normal
1	00313ee0-9eaa-42f4-b0ab-c148ed3241cd	No Lung Opacity / Not Normal
2	00322d4d-1c29-4943-afc9-b6754be640eb	No Lung Opacity / Not Normal
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	Normal
4	00436515-870c-4b36-a041-de91049b9ab4	Lung Opacity

def merge_data_frames(left_df,right_df,merge_on):
  df = pd.merge(left = left_df, right = right_df, how = 'left', on = merge_on)
  df = df.drop_duplicates()
  df.info()

merge_data_frames(class_desc,labels,'patientId')

<class 'pandas.core.frame.DataFrame'>
Int64Index: 30227 entries, 0 to 37626
Data columns (total 7 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   patientId  30227 non-null  object 
 1   class      30227 non-null  object 
 2   x          9555 non-null   float64
 3   y          9555 non-null   float64
 4   width      9555 non-null   float64
 5   height     9555 non-null   float64
 6   Target     30227 non-null  int64  
dtypes: float64(4), int64(1), object(2)
memory usage: 1.8+ MB

class_desc.dtypes

patientId    object
class        object
dtype: object

print("Number of unique patient IDs: ", class_desc['patientId'].nunique())

Number of unique patient IDs:  26684

class_desc['patientId'].value_counts().value_counts()

1    23286
2     3266
3      119
4       13
Name: patientId, dtype: int64

labels.shape

(30227, 6)

labels.dtypes

patientId     object
x            float64
y            float64
width        float64
height       float64
Target         int64
dtype: object

labels.isnull().values.any()

True

labels = labels.fillna(0)
labels.head(6)

	patientId	x	y	width	height	Target
0	0004cfab-14fd-4e49-80ba-63a80b6bddd6	0.0	0.0	0.0	0.0	0
1	00313ee0-9eaa-42f4-b0ab-c148ed3241cd	0.0	0.0	0.0	0.0	0
2	00322d4d-1c29-4943-afc9-b6754be640eb	0.0	0.0	0.0	0.0	0
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	0.0	0.0	0.0	0.0	0
4	00436515-870c-4b36-a041-de91049b9ab4	264.0	152.0	213.0	379.0	1
5	00436515-870c-4b36-a041-de91049b9ab4	562.0	152.0	256.0	453.0	1

Database table Preprocessing

def check_for_missing_data(df):
    total = df.isnull().sum().sort_values(ascending=False)   # finding total number of null values
    percent = (df.isnull().sum()/df.isnull().count()).sort_values(ascending=False) #percentage of values that are null
    missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent']) # putting the above two together
    return missing_data

check_for_missing_data(labels)

	Total	Percent
patientId	0	0.0
x	0	0.0
y	0	0.0
width	0	0.0
height	0	0.0
Target	0	0.0

check_for_missing_data(class_desc)

	Total	Percent
patientId	0	0.0
class	0	0.0

Observation : No missing value found in the class_info table. All the target values are filled therefore there is no empty value, cell with NaN or empty cell in label_data are having no phenumonia so let's replace those value with 0.

labels.fillna(0, inplace=True)

labels.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 30227 entries, 0 to 30226
Data columns (total 6 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   patientId  30227 non-null  object 
 1   x          30227 non-null  float64
 2   y          30227 non-null  float64
 3   width      30227 non-null  float64
 4   height     30227 non-null  float64
 5   Target     30227 non-null  int64  
dtypes: float64(4), int64(1), object(1)
memory usage: 1.4+ MB

labels.head()

	patientId	x	y	width	height	Target
0	0004cfab-14fd-4e49-80ba-63a80b6bddd6	0.0	0.0	0.0	0.0	0
1	00313ee0-9eaa-42f4-b0ab-c148ed3241cd	0.0	0.0	0.0	0.0	0
2	00322d4d-1c29-4943-afc9-b6754be640eb	0.0	0.0	0.0	0.0	0
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	0.0	0.0	0.0	0.0	0
4	00436515-870c-4b36-a041-de91049b9ab4	264.0	152.0	213.0	379.0	1

labels[labels["Target"]==1].describe().T

	count	mean	std	min	25%	50%	75%	max
x	9555.0	394.047724	204.574172	2.0	207.0	324.0	594.0	835.0
y	9555.0	366.839560	148.940488	2.0	249.0	365.0	478.5	881.0
width	9555.0	218.471376	59.289475	40.0	177.0	217.0	259.0	528.0
height	9555.0	329.269702	157.750755	45.0	203.0	298.0	438.0	942.0
Target	9555.0	1.000000	0.000000	1.0	1.0	1.0	1.0	1.0

np.unique(class_desc["class"])  

array(['Lung Opacity', 'No Lung Opacity / Not Normal', 'Normal'],
      dtype=object)

There are 3 classes -

1. Lung Opacity
2. No Lung Opacity / Not Normal
3. Normal

#Let us plot the distribution
plt.rcParams['figure.figsize'] = [10, 4]
plt.rcParams['lines.linewidth'] = 2

def countPlot(target="Target"):
    ax = sns.countplot(x=target, data=labels, palette="pastel")

    for p in ax.patches:
        ax.annotate('{}'.format(p.get_height()), (p.get_x()+0.400, p.get_height()+0.20), ha='center')

    plt.show()

countPlot("Target")

#To plot the 3 categories of the class
f, ax = plt.subplots(1,1, figsize=(14,10))
total = float(len(class_desc))
sns.countplot(class_desc['class'],order = class_desc['class'].value_counts().index, palette='Set3')
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x()+p.get_width()/2.,
            height + 6,
            '{:1.2f}%'.format(100*height/total),
            ha="center") 
plt.show()

class_desc[class_desc["patientId"]=='0004cfab-14fd-4e49-80ba-63a80b6bddd6']

	patientId	class
0	0004cfab-14fd-4e49-80ba-63a80b6bddd6	No Lung Opacity / Not Normal

labels[labels['patientId']=='0004cfab-14fd-4e49-80ba-63a80b6bddd6']

	patientId	x	y	width	height	Target
0	0004cfab-14fd-4e49-80ba-63a80b6bddd6	0.0	0.0	0.0	0.0	0

No Lung Opacity / Not Normal has been classified as Target 0

class_desc[class_desc["patientId"]=='003d8fa0-6bf1-40ed-b54c-ac657f8495c5']

	patientId	class
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	Normal

labels[labels['patientId']=='003d8fa0-6bf1-40ed-b54c-ac657f8495c5']

	patientId	x	y	width	height	Target
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	0.0	0.0	0.0	0.0	0

Normal has been classified as 0 as well

class_desc[class_desc["patientId"]=='00436515-870c-4b36-a041-de91049b9ab4']

	patientId	class
4	00436515-870c-4b36-a041-de91049b9ab4	Lung Opacity
5	00436515-870c-4b36-a041-de91049b9ab4	Lung Opacity

labels[labels['patientId']=='00436515-870c-4b36-a041-de91049b9ab4']

	patientId	x	y	width	height	Target
4	00436515-870c-4b36-a041-de91049b9ab4	264.0	152.0	213.0	379.0	1
5	00436515-870c-4b36-a041-de91049b9ab4	562.0	152.0	256.0	453.0	1

Lung Opacity has been classified as 1

!mkdir -p /content/sample_data/Stage2-data

os.chdir('/content/sample_data/Stage2-data/')

!pwd

!ls -lart '/content/sample_data/Stage2-data/'

/content/sample_data/Stage2-data
total 8
drwxr-xr-x 1 root root 4096 Mar  4 14:37 ..
drwxr-xr-x 2 root root 4096 Mar  4 14:37 .

from zipfile import ZipFile
import os

# Create a ZipFile Object and load sample.zip in it
def extract_zipFile(source_path,destination_path):
  try:
    with ZipFile(source_path, 'r') as zipObj:
      # Extract all the contents of zip file in destination_path
      zipObj.extractall(destination_path)
  except Exception as e:
    print(str(e))

#extracting train images
print("Extracting train images...")
extract_zipFile('/content/drive/MyDrive/stage_2_train_images.zip','/content/sample_data/Stage2-data/stage_2_train_images')

#extracting test images
print("Extracting test images...")
extract_zipFile('/content/drive/MyDrive/stage_2_test_images.zip','/content/sample_data/Stage2-data/stage_2_test_images')

Extracting train images...
Extracting test images...

# Images Example
train_images_folder = "/".join(('/content/sample_data/Stage2-data', "stage_2_train_images/stage_2_train_images/"))
test_images_folder = "/".join(('/content/sample_data/Stage2-data', "stage_2_test_images/stage_2_test_images/"))
def parse_dicom_images(image_dir):
  images_list = []
  images = os.listdir(image_dir)
  for image in images:
    image_file = os.path.join(image_dir, image)
    if os.path.isfile(image_file):
      images_list.append(image_file)
  return images_list

train_images = parse_dicom_images(train_images_folder)
test_images = parse_dicom_images(test_images_folder)
print('5 Training images', train_images[:5])
print('5 Test images', test_images[:5])

5 Training images ['/content/sample_data/Stage2-data/stage_2_train_images/stage_2_train_images/fddfb799-68be-418d-8caf-c99045137eb8.dcm', '/content/sample_data/Stage2-data/stage_2_train_images/stage_2_train_images/710fbf77-85ac-4974-9077-0d4d9fdb3e48.dcm', '/content/sample_data/Stage2-data/stage_2_train_images/stage_2_train_images/d01c47aa-347c-4919-a82e-b90b7d3b164f.dcm', '/content/sample_data/Stage2-data/stage_2_train_images/stage_2_train_images/f943bdb9-3c23-49f4-be0b-0f8bff48c7d3.dcm', '/content/sample_data/Stage2-data/stage_2_train_images/stage_2_train_images/7bb751e1-6d7d-4640-b48e-fb1ea39b0ad9.dcm']
5 Test images ['/content/sample_data/Stage2-data/stage_2_test_images/stage_2_test_images/248be5cb-5079-4ba7-b2ba-a0ff1062fff2.dcm', '/content/sample_data/Stage2-data/stage_2_test_images/stage_2_test_images/05b2cc6d-8a25-47c8-a191-bbaaadf0bf64.dcm', '/content/sample_data/Stage2-data/stage_2_test_images/stage_2_test_images/1a664216-c58a-4405-bb4b-7177d919e53d.dcm', '/content/sample_data/Stage2-data/stage_2_test_images/stage_2_test_images/c0c629fa-2d12-4534-a792-a70e1f6c967e.dcm', '/content/sample_data/Stage2-data/stage_2_test_images/stage_2_test_images/c127904f-d321-4d79-b02d-599b73b0a734.dcm']

column_list = ["Patient ID", "Patient Sex", "Patient's Age", "View Position", "Image Size"]
file_desc = pd.DataFrame(columns=column_list)

def add_images_to_df(df,images_list,count_limit=100):
    data = []
    count = 0
    for image_path in images_list:
      data_row_img_data = dcm.read_file(image_path)
      values = []
      count = count + 1
      values.append(data_row_img_data.PatientID)
      values.append(data_row_img_data.PatientSex)
      values.append(data_row_img_data.PatientAge)
      values.append(data_row_img_data.ViewPosition)
      values.append(f"{data_row_img_data.Rows}x{data_row_img_data.Columns}")
      zipped_val = dict(zip(column_list, values))
      df = df.append(zipped_val, True)
    print(df)
    return df

file_desc = add_images_to_df(file_desc, train_images,10000)

                                 Patient ID Patient Sex Patient's Age  \
0      fddfb799-68be-418d-8caf-c99045137eb8           M            46   
1      710fbf77-85ac-4974-9077-0d4d9fdb3e48           M            52   
2      d01c47aa-347c-4919-a82e-b90b7d3b164f           M            41   
3      f943bdb9-3c23-49f4-be0b-0f8bff48c7d3           F            52   
4      7bb751e1-6d7d-4640-b48e-fb1ea39b0ad9           M            79   
...                                     ...         ...           ...   
26679  54d69a5e-7f51-46fc-b141-0fd7e7086fae           M            50   
26680  817c90f1-a049-474e-bfd5-4e937e6fc234           M            68   
26681  a3b6bc70-8f6b-42b2-8a93-2b5676270ab6           F            34   
26682  8d837773-a674-4af2-b401-2b3216586c5e           M            45   
26683  9cc13090-1906-4f88-83e1-bde1c174bbec           F            68   

      View Position Image Size  
0                PA  1024x1024  
1                PA  1024x1024  
2                PA  1024x1024  
3                PA  1024x1024  
4                PA  1024x1024  
...             ...        ...  
26679            PA  1024x1024  
26680            PA  1024x1024  
26681            PA  1024x1024  
26682            AP  1024x1024  
26683            PA  1024x1024  

[26684 rows x 5 columns]

file_desc.info()

<class 'pandas.core.frame.DataFrame'>
Index: 0 entries
Data columns (total 5 columns):
 #   Column         Non-Null Count  Dtype 
---  ------         --------------  ----- 
 0   Patient ID     0 non-null      object
 1   Patient Sex    0 non-null      object
 2   Patient's Age  0 non-null      object
 3   View Position  0 non-null      object
 4   Image Size     0 non-null      object
dtypes: object(5)
memory usage: 0.0+ bytes

train_images_folder

'/content/sample_data/Stage2-data/stage_2_train_images/stage_2_train_images/'

dicom_file_path = os.path.join(train_images_folder + "05eebe4c-bca2-40d4-bb20-54fc60e2bcea.dcm")
file = dcm.read_file(dicom_file_path)
file

Dataset.file_meta -------------------------------
(0002, 0000) File Meta Information Group Length  UL: 202
(0002, 0001) File Meta Information Version       OB: b'\x00\x01'
(0002, 0002) Media Storage SOP Class UID         UI: Secondary Capture Image Storage
(0002, 0003) Media Storage SOP Instance UID      UI: 1.2.276.0.7230010.3.1.4.8323329.15214.1517874384.336739
(0002, 0010) Transfer Syntax UID                 UI: JPEG Baseline (Process 1)
(0002, 0012) Implementation Class UID            UI: 1.2.276.0.7230010.3.0.3.6.0
(0002, 0013) Implementation Version Name         SH: 'OFFIS_DCMTK_360'
-------------------------------------------------
(0008, 0005) Specific Character Set              CS: 'ISO_IR 100'
(0008, 0016) SOP Class UID                       UI: Secondary Capture Image Storage
(0008, 0018) SOP Instance UID                    UI: 1.2.276.0.7230010.3.1.4.8323329.15214.1517874384.336739
(0008, 0020) Study Date                          DA: '19010101'
(0008, 0030) Study Time                          TM: '000000.00'
(0008, 0050) Accession Number                    SH: ''
(0008, 0060) Modality                            CS: 'CR'
(0008, 0064) Conversion Type                     CS: 'WSD'
(0008, 0090) Referring Physician's Name          PN: ''
(0008, 103e) Series Description                  LO: 'view: AP'
(0010, 0010) Patient's Name                      PN: '05eebe4c-bca2-40d4-bb20-54fc60e2bcea'
(0010, 0020) Patient ID                          LO: '05eebe4c-bca2-40d4-bb20-54fc60e2bcea'
(0010, 0030) Patient's Birth Date                DA: ''
(0010, 0040) Patient's Sex                       CS: 'F'
(0010, 1010) Patient's Age                       AS: '56'
(0018, 0015) Body Part Examined                  CS: 'CHEST'
(0018, 5101) View Position                       CS: 'AP'
(0020, 000d) Study Instance UID                  UI: 1.2.276.0.7230010.3.1.2.8323329.15214.1517874384.336738
(0020, 000e) Series Instance UID                 UI: 1.2.276.0.7230010.3.1.3.8323329.15214.1517874384.336737
(0020, 0010) Study ID                            SH: ''
(0020, 0011) Series Number                       IS: '1'
(0020, 0013) Instance Number                     IS: '1'
(0020, 0020) Patient Orientation                 CS: ''
(0028, 0002) Samples per Pixel                   US: 1
(0028, 0004) Photometric Interpretation          CS: 'MONOCHROME2'
(0028, 0010) Rows                                US: 1024
(0028, 0011) Columns                             US: 1024
(0028, 0030) Pixel Spacing                       DS: [0.168, 0.168]
(0028, 0100) Bits Allocated                      US: 8
(0028, 0101) Bits Stored                         US: 8
(0028, 0102) High Bit                            US: 7
(0028, 0103) Pixel Representation                US: 0
(0028, 2110) Lossy Image Compression             CS: '01'
(0028, 2114) Lossy Image Compression Method      CS: 'ISO_10918_1'
(7fe0, 0010) Pixel Data                          OB: Array of 103170 elements

Data Visualisation

gc.collect()

355

def show_dicom_images_with_boxes(data):
    img_data = list(data.T.to_dict().values())
    f, ax = plt.subplots(3,3, figsize=(16,18))
    for i,data_row in enumerate(img_data):
        patientImage = data_row['patientId']+'.dcm'
        imagePath = os.path.join("",train_images_folder,patientImage)
        data_row_img_data = dcm.read_file(imagePath)
        modality = data_row_img_data.Modality
        age = data_row_img_data.PatientAge
        sex = data_row_img_data.PatientSex
        data_row_img = dcm.dcmread(imagePath)
        ax[i//3, i%3].imshow(data_row_img.pixel_array, cmap=plt.cm.bone) 
        ax[i//3, i%3].axis('off')
        ax[i//3, i%3].set_title('ID: {}\nModality: {} Age: {} Sex: {} Target:'.format(
                data_row['patientId'],modality, age, sex, data_row['Target']))
        rows = labels[labels['patientId']==data_row['patientId']]
        box_data = list(rows.T.to_dict().values())
        for j, row in enumerate(box_data):
            ax[i//3, i%3].add_patch(Rectangle(xy=(row['x'], row['y']),
                        width=row['width'],height=row['height'], 
                        color="red",alpha = 0.1))   
    plt.show()

target = labels[labels['Target']==1]
print(type(target))
print(target.head())
#generating 9 row
show_dicom_images_with_boxes(labels[labels['Target']==1].sample(9))

<class 'pandas.core.frame.DataFrame'>
                               patientId      x      y  width  height  Target
4   00436515-870c-4b36-a041-de91049b9ab4  264.0  152.0  213.0   379.0       1
5   00436515-870c-4b36-a041-de91049b9ab4  562.0  152.0  256.0   453.0       1
8   00704310-78a8-4b38-8475-49f4573b2dbb  323.0  577.0  160.0   104.0       1
9   00704310-78a8-4b38-8475-49f4573b2dbb  695.0  575.0  162.0   137.0       1
14  00aecb01-a116-45a2-956c-08d2fa55433f  288.0  322.0   94.0   135.0       1

show_dicom_images_with_boxes(labels[labels['Target']==0].sample(9))

def show_dicom_images(data):
    img_data = list(data.T.to_dict().values())
    f, ax = plt.subplots(3,3, figsize=(16,18))
    for i,data_row in enumerate(img_data):
        patientImage = data_row['patientId']+'.dcm'
        imagePath = os.path.join(train_images_folder,patientImage)
        data_row_img_data = dcm.read_file(imagePath)
        modality = data_row_img_data.Modality
        age = data_row_img_data.PatientAge
        sex = data_row_img_data.PatientSex
        data_row_img = dcm.dcmread(imagePath)
        ax[i//3, i%3].imshow(data_row_img.pixel_array, cmap=plt.cm.bone) 
        ax[i//3, i%3].axis('off')
        ax[i//3, i%3].set_title('ID: {}\nModality: {} Age: {} Sex: {} Target: \nWindow: {}:{}:{}:{}'.format(
                data_row['patientId'],
                modality, age, sex, data_row['Target'], 
                data_row['x'],data_row['y'],data_row['width'],data_row['height']))
    plt.show()
    return data_row_img_data

data_row_img_data = show_dicom_images(labels[labels['Target']==1].sample(9))

boxes_per_patient = labels.groupby('patientId')['Target'].sum()

ax = (boxes_per_patient > 0).value_counts().plot.barh(color=['blue','orange'])
_ = ax.set_title('Pneumonia opacity present')
_ = ax.set_xlabel('Number of patients')
_ = ax.xaxis.set_tick_params(rotation=0)

import seaborn as sns
sns.set(color_codes=True)

centers = (labels.dropna(subset=['x'])
           .assign(center_x=labels.x + labels.width / 2, 
                   center_y=labels.y + labels.height / 2))
ax = sns.jointplot("center_x", "center_y", data=centers, height=6, alpha=0.03, color="green")
_ = ax.fig.suptitle("Where is Pneumonia located?", y=1.01)

labels['aspect_ratio'] = (labels['width'] / labels['height'])
labels['area'] = labels['width'] * labels['height']

ax = sns.distplot(labels['aspect_ratio'].dropna(), norm_hist=True)
_ = ax.set_title("Distribution of bounding aspect ratios")
_ = ax.set_xlabel("Aspect Ratio")

sns.distplot(labels['x'],kde=True,bins=50, color="violet")
sns.distplot(labels['y'],kde=True,bins=50, color="red")
sns.distplot(labels['width'],kde=True,bins=50, color="green")
sns.distplot(labels['height'],kde=True,bins=50, color="yellow")

<matplotlib.axes._subplots.AxesSubplot at 0x7f3b08d142d0>

import multiprocessing

def get_info(patientId, root_dir=train_images_folder):
    file_name = os.path.join(root_dir, f'{patientId}.dcm')
    dicom_data = dcm.read_file(file_name)
    return {'age': dicom_data.PatientAge, 
            'gender': dicom_data.PatientSex,
            'view_position': dicom_data.ViewPosition,
            'id': os.path.basename(file_name).split('.')[0],
            'pixel_spacing': float(dicom_data.PixelSpacing[0]),
            'mean_black_pixels': np.mean(dicom_data.pixel_array == 0)}

patient_ids = list(labels.patientId.unique())
with multiprocessing.Pool(4) as pool:
    result = pool.map(get_info, patient_ids)
    
demo = pd.DataFrame(result)
demo['age'] = demo['age'].astype(int)
demo['gender'] = demo['gender'].astype('category')
demo['view_position'] = demo['view_position'].astype('category')

labels = (labels.merge(demo, left_on='patientId', right_on='id', how='left').drop(columns='id'))

labels[2:5]

	patientId	x	y	width	height	Target	aspect_ratio	area	age	gender	view_position	pixel_spacing	mean_black_pixels
2	00322d4d-1c29-4943-afc9-b6754be640eb	0.0	0.0	0.0	0.0	0	NaN	0.0	19	M	AP	0.168	0.000000
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	0.0	0.0	0.0	0.0	0	NaN	0.0	28	M	PA	0.143	0.011372
4	00436515-870c-4b36-a041-de91049b9ab4	264.0	152.0	213.0	379.0	1	0.562005	80727.0	32	F	AP	0.139	0.002634

import seaborn as sns
sns.set(color_codes=True)

g = sns.FacetGrid(col='Target', hue='gender', 
                  data=labels.drop_duplicates(subset=['patientId']), 
                  height=9, palette=dict(F="red", M="blue"))
_ = g.map(sns.distplot, 'age', hist_kws={'alpha': 0.5}).add_legend()
_ = g.fig.suptitle("Age distribution by gender and target?", y=1.02, fontsize=20)

VGG-16

from tensorflow.keras.applications import VGG16
from tensorflow.keras.layers import Flatten, Dense, Input, Concatenate
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam, SGD
from sklearn.model_selection import train_test_split
import numpy as np
import cv2
import os
import pydicom as dcm
import pandas as pd
import pickle

data = []
bbox_coords = []
target_label = []
img_paths = []

label_data=labels[0:10000]

label_data[2:5]

	patientId	x	y	width	height	Target	aspect_ratio	area	age	gender	view_position	pixel_spacing	mean_black_pixels
2	00322d4d-1c29-4943-afc9-b6754be640eb	0.0	0.0	0.0	0.0	0	NaN	0.0	19	M	AP	0.168	0.000000
3	003d8fa0-6bf1-40ed-b54c-ac657f8495c5	0.0	0.0	0.0	0.0	0	NaN	0.0	28	M	PA	0.143	0.011372
4	00436515-870c-4b36-a041-de91049b9ab4	264.0	152.0	213.0	379.0	1	0.562005	80727.0	32	F	AP	0.139	0.002634

for i in range(len(label_data)):
  filename=label_data.iloc[i,0]
  startX = label_data.iloc[i,1]
  startY = label_data.iloc[i,2]
  endX = label_data.iloc[i,3]
  endY = label_data.iloc[i,4]
  label= label_data.iloc[i,5]
  img_path = os.path.join('/content/sample_data/Stage2-data/stage_2_train_images/stage_2_train_images/', filename+".dcm")##Change the path to your directory where images are located
  img = dcm.dcmread(img_path).pixel_array

  h = float(img.shape[0])
  w = float(img.shape[1])

  # Scale bounding box coordinates relative to the image
  startX = float(startX)/w
  startY = float(startY)/h
  endX = float(endX)/w
  endY = float(endY)/h
  img = cv2.resize(img,(224,224))
  data.append(img)
  bbox_coords.append((startX,startY,endX,endY))
  target_label.append(label)
  img_paths.append(img_path)

#Scaling data and bbox
data = np.array(data, dtype="float32") / 255.0
target_label = np.array(target_label)
bbox_coords = np.array(bbox_coords, dtype="float32")

# Saving data to disk
with open(os.path.join("/content/drive/MyDrive/","data.pkl"), 'wb') as f:
    pickle.dump(data, f, protocol=4)

with open(os.path.join("/content/drive/MyDrive/","target_label.pkl"),'wb') as f:
    pickle.dump(target_label, f, protocol=4)

with open(os.path.join("/content/drive/MyDrive/","bbox_coords.pkl"),'wb') as f:
    pickle.dump(bbox_coords, f, protocol=4)

with open(os.path.join("/content/drive/MyDrive/","img_paths.pkl"),'wb') as f:
    pickle.dump(img_paths, f, protocol=4)

# Fine tuning VGG16
image_input = Input(shape=(224,224,1))
image_input = Concatenate()([image_input,image_input,image_input])
vgg_model = VGG16(weights="imagenet", include_top=False,input_tensor= image_input)
vgg_model.trainable = False

basemodel_output = vgg_model.output
flatten = Flatten()(basemodel_output)

# New FC layers for Regression
bboxHead = Dense(128, activation="relu")(flatten)
bboxHead = Dense(64, activation="relu")(bboxHead)
bboxHead = Dense(32, activation="relu")(bboxHead)
bboxHead = Dense(4, activation="sigmoid", name="BBOX_Head")(bboxHead)

# New FC layers for Classification
classHead = Dense(512, activation="relu")(flatten)
classHead = Dense(512, activation="relu")(classHead)
classHead = Dense(128, activation="relu")(classHead)
classHead = Dense(1, activation="sigmoid", name="Classification_Head")(classHead)

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5
58892288/58889256 [==============================] - 1s 0us/step
58900480/58889256 [==============================] - 1s 0us/step

model = Model(inputs=vgg_model.input, outputs=(classHead, bboxHead))

opt = SGD(lr=0.01)
losses = {
    "Classification_Head": "categorical_crossentropy",
    "BBOX_Head": "mean_squared_error"
}

loss_weights = {
    "Classification_Head": 1.0,
    "BBOX_Head": 1.0
}

model.compile(loss=losses, optimizer=opt, loss_weights= loss_weights, metrics=["accuracy"])

print(model.summary())

Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_1 (InputLayer)           [(None, 224, 224, 1  0           []                               
                                )]                                                                
                                                                                                  
 concatenate (Concatenate)      (None, 224, 224, 3)  0           ['input_1[0][0]',                
                                                                  'input_1[0][0]',                
                                                                  'input_1[0][0]']                
                                                                                                  
 block1_conv1 (Conv2D)          (None, 224, 224, 64  1792        ['concatenate[0][0]']            
                                )                                                                 
                                                                                                  
 block1_conv2 (Conv2D)          (None, 224, 224, 64  36928       ['block1_conv1[0][0]']           
                                )                                                                 
                                                                                                  
 block1_pool (MaxPooling2D)     (None, 112, 112, 64  0           ['block1_conv2[0][0]']           
                                )                                                                 
                                                                                                  
 block2_conv1 (Conv2D)          (None, 112, 112, 12  73856       ['block1_pool[0][0]']            
                                8)                                                                
                                                                                                  
 block2_conv2 (Conv2D)          (None, 112, 112, 12  147584      ['block2_conv1[0][0]']           
                                8)                                                                
                                                                                                  
 block2_pool (MaxPooling2D)     (None, 56, 56, 128)  0           ['block2_conv2[0][0]']           
                                                                                                  
 block3_conv1 (Conv2D)          (None, 56, 56, 256)  295168      ['block2_pool[0][0]']            
                                                                                                  
 block3_conv2 (Conv2D)          (None, 56, 56, 256)  590080      ['block3_conv1[0][0]']           
                                                                                                  
 block3_conv3 (Conv2D)          (None, 56, 56, 256)  590080      ['block3_conv2[0][0]']           
                                                                                                  
 block3_pool (MaxPooling2D)     (None, 28, 28, 256)  0           ['block3_conv3[0][0]']           
                                                                                                  
 block4_conv1 (Conv2D)          (None, 28, 28, 512)  1180160     ['block3_pool[0][0]']            
                                                                                                  
 block4_conv2 (Conv2D)          (None, 28, 28, 512)  2359808     ['block4_conv1[0][0]']           
                                                                                                  
 block4_conv3 (Conv2D)          (None, 28, 28, 512)  2359808     ['block4_conv2[0][0]']           
                                                                                                  
 block4_pool (MaxPooling2D)     (None, 14, 14, 512)  0           ['block4_conv3[0][0]']           
                                                                                                  
 block5_conv1 (Conv2D)          (None, 14, 14, 512)  2359808     ['block4_pool[0][0]']            
                                                                                                  
 block5_conv2 (Conv2D)          (None, 14, 14, 512)  2359808     ['block5_conv1[0][0]']           
                                                                                                  
 block5_conv3 (Conv2D)          (None, 14, 14, 512)  2359808     ['block5_conv2[0][0]']           
                                                                                                  
 block5_pool (MaxPooling2D)     (None, 7, 7, 512)    0           ['block5_conv3[0][0]']           
                                                                                                  
 flatten (Flatten)              (None, 25088)        0           ['block5_pool[0][0]']            
                                                                                                  
 dense_3 (Dense)                (None, 512)          12845568    ['flatten[0][0]']                
                                                                                                  
 dense (Dense)                  (None, 128)          3211392     ['flatten[0][0]']                
                                                                                                  
 dense_4 (Dense)                (None, 512)          262656      ['dense_3[0][0]']                
                                                                                                  
 dense_1 (Dense)                (None, 64)           8256        ['dense[0][0]']                  
                                                                                                  
 dense_5 (Dense)                (None, 128)          65664       ['dense_4[0][0]']                
                                                                                                  
 dense_2 (Dense)                (None, 32)           2080        ['dense_1[0][0]']                
                                                                                                  
 Classification_Head (Dense)    (None, 1)            129         ['dense_5[0][0]']                
                                                                                                  
 BBOX_Head (Dense)              (None, 4)            132         ['dense_2[0][0]']                
                                                                                                  
==================================================================================================
Total params: 31,110,565
Trainable params: 16,395,877
Non-trainable params: 14,714,688
__________________________________________________________________________________________________
None

from keras.utils.vis_utils import plot_model
import tensorflow as tf
tf.keras.utils.plot_model(
model, to_file='model.png', show_shapes=False, show_dtype=False,
show_layer_names=True, rankdir='TB', expand_nested=False, dpi=96
)

data_split = train_test_split(data, target_label, bbox_coords, img_paths,	test_size=0.30, random_state=12)

(trainImages, testImages) = data_split[:2]
(trainLabels, testLabels) = data_split[2:4]
(trainBBoxes, testBBoxes) = data_split[4:6]
(trainPaths, testPaths) = data_split[6:]

print("[INFO] Training the Model...")

History = model.fit(
    trainImages, [trainLabels, trainBBoxes],
	validation_data=(testImages, [testLabels,testBBoxes]),
	batch_size=32,
	epochs=20,
	verbose=1)

[INFO] Training the Model...
Epoch 1/20
219/219 [==============================] - 115s 437ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0307 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3421 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0279 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.4927
Epoch 2/20
219/219 [==============================] - 92s 419ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0278 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.4440 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0266 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3587
Epoch 3/20
219/219 [==============================] - 91s 417ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0266 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3849 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0256 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3807
Epoch 4/20
219/219 [==============================] - 91s 417ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0259 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3540 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0255 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.4687
Epoch 5/20
219/219 [==============================] - 91s 417ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0254 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3451 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0245 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3533
Epoch 6/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0249 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3261 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0252 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3817
Epoch 7/20
219/219 [==============================] - 91s 417ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0246 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3157 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0238 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3317
Epoch 8/20
219/219 [==============================] - 91s 417ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0243 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3229 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0237 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.2983
Epoch 9/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0240 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3081 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0234 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3057
Epoch 10/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0238 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3049 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0245 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.4160
Epoch 11/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0236 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3030 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0231 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.2460
Epoch 12/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0235 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.2993 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0231 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.2583
Epoch 13/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0234 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.2991 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0231 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3390
Epoch 14/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0232 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3047 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0231 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3607
Epoch 15/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0231 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.2901 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0229 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.1930
Epoch 16/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0231 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.2943 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0226 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.2970
Epoch 17/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0229 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.2986 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0227 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.2933
Epoch 18/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0227 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.2936 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0233 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.3077
Epoch 19/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0227 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3037 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0224 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.2547
Epoch 20/20
219/219 [==============================] - 91s 416ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: 0.0226 - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.3001 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: 0.0223 - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.2577

# Save model
print("[INFO] Saving Object Detector Model...")
model.save("/content/drive/MyDrive/VGGMOD", save_format="h5")#Use your drive path

[INFO] Saving Object Detector Model...

# plot the loss
import matplotlib.pyplot as plt
plt.plot(History.history['Classification_Head_loss'], label='train Classification_Head_loss')
plt.plot(History.history['val_Classification_Head_loss'], label='val_Classification_Head_loss')
plt.plot(History.history['BBOX_Head_loss'], label='train BBOX_Head_loss')
plt.plot(History.history['val_BBOX_Head_loss'], label='val BBOX_Head_loss')

plt.legend()
plt.show()
plt.savefig('LossVal_loss')

# plot the accuracy
plt.plot(History.history['Classification_Head_accuracy'], label='train Classification_Head_accuracy')
plt.plot(History.history['val_Classification_Head_accuracy'], label='val_Classification_Head_accuracy')
plt.plot(History.history['BBOX_Head_accuracy'], label='train BBOX_Head_accuracy')
plt.plot(History.history['val_BBOX_Head_accuracy'], label='val_BBOX_Head_accuracy')
plt.legend()
plt.show()
plt.savefig('AccVal_acc')

<Figure size 720x288 with 0 Axes>

Without a Pre-Trained Model

import tensorflow as tf
from tensorflow.keras.regularizers import l2
from tensorflow.keras.layers import Conv2D,MaxPool2D

# Part 2 - Building the CNN
# Initialising the CNN
cnn = tf.keras.models.Sequential()

# Step 1 - Convolution
cnn.add(tf.keras.layers.Conv2D(filters=32,padding="same",kernel_size=3, activation='relu', strides=2, input_shape=[224, 224, 1]))

# Step 2 - Pooling
cnn.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))

# Adding a second convolutional layer
cnn.add(tf.keras.layers.Conv2D(filters=32,padding='same',kernel_size=3, activation='relu'))
cnn.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))

# Step 3 - Flattening
cnn.add(tf.keras.layers.Flatten())


x0 = cnn.layers[-1].output



# New FC layers for Regression
bboxHead = Dense(128, activation="relu")(x0)
bboxHead = Dense(64, activation="relu")(bboxHead)
bboxHead = Dense(32, activation="relu")(bboxHead)
bboxHead = Dense(4, activation="sigmoid", name="BBOX_Head")(bboxHead)

# New FC layers for Classification
classHead = Dense(512, activation="relu")(x0)
classHead = Dense(256, activation="relu")(classHead)
classHead = Dense(128, activation="relu")(classHead)
classHead = Dense(1, activation="sigmoid", name="Classification_Head")(classHead)


# # Step 4 - Full Connection
cnn.add(tf.keras.layers.Dense(units=128, activation='relu'))

# # Step 5 - Output Layer
cnn.add(tf.keras.layers.Dense(units=1, activation='sigmoid'))
# ## For Binary Classification
cnn.add(Dense(1, kernel_regularizer=tf.keras.regularizers.l2(0.01),activation
='linear'))

tf.keras.utils.plot_model(
cnn, to_file='cnn.png', show_shapes=False, show_dtype=False,
show_layer_names=True, rankdir='TB', expand_nested=False, dpi=96
)

#cnn.compile(optimizer = 'adam', loss = 'squared_hinge', metrics = ['accuracy'])
cnn.compile(loss=losses, optimizer=opt, loss_weights= loss_weights, metrics=["accuracy"])

cnn.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 112, 112, 32)      320       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 56, 56, 32)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 56, 56, 32)        9248      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 28, 28, 32)       0         
 2D)                                                             
                                                                 
 flatten_1 (Flatten)         (None, 25088)             0         
                                                                 
 dense_12 (Dense)            (None, 128)               3211392   
                                                                 
 dense_13 (Dense)            (None, 1)                 129       
                                                                 
 dense_14 (Dense)            (None, 1)                 2         
                                                                 
=================================================================
Total params: 3,221,091
Trainable params: 3,221,091
Non-trainable params: 0
_________________________________________________________________

model = Model(inputs=cnn.input, outputs=(classHead, bboxHead))

model.compile(loss=losses, optimizer=opt, loss_weights= loss_weights, metrics=["accuracy"])

print(model.summary())

Model: "model_1"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 conv2d_input (InputLayer)      [(None, 224, 224, 1  0           []                               
                                )]                                                                
                                                                                                  
 conv2d (Conv2D)                (None, 112, 112, 32  320         ['conv2d_input[0][0]']           
                                )                                                                 
                                                                                                  
 max_pooling2d (MaxPooling2D)   (None, 56, 56, 32)   0           ['conv2d[0][0]']                 
                                                                                                  
 conv2d_1 (Conv2D)              (None, 56, 56, 32)   9248        ['max_pooling2d[0][0]']          
                                                                                                  
 max_pooling2d_1 (MaxPooling2D)  (None, 28, 28, 32)  0           ['conv2d_1[0][0]']               
                                                                                                  
 flatten_1 (Flatten)            (None, 25088)        0           ['max_pooling2d_1[0][0]']        
                                                                                                  
 dense_9 (Dense)                (None, 512)          12845568    ['flatten_1[0][0]']              
                                                                                                  
 dense_6 (Dense)                (None, 128)          3211392     ['flatten_1[0][0]']              
                                                                                                  
 dense_10 (Dense)               (None, 256)          131328      ['dense_9[0][0]']                
                                                                                                  
 dense_7 (Dense)                (None, 64)           8256        ['dense_6[0][0]']                
                                                                                                  
 dense_11 (Dense)               (None, 128)          32896       ['dense_10[0][0]']               
                                                                                                  
 dense_8 (Dense)                (None, 32)           2080        ['dense_7[0][0]']                
                                                                                                  
 Classification_Head (Dense)    (None, 1)            129         ['dense_11[0][0]']               
                                                                                                  
 BBOX_Head (Dense)              (None, 4)            132         ['dense_8[0][0]']                
                                                                                                  
==================================================================================================
Total params: 16,241,349
Trainable params: 16,241,349
Non-trainable params: 0
__________________________________________________________________________________________________
None

print("[INFO] Training the Model...")

History = model.fit(
    trainImages, [trainLabels, trainBBoxes],
	validation_data=(testImages, [testLabels,testBBoxes]),
	batch_size=32,
	epochs=20,
	verbose=1)

[INFO] Training the Model...
Epoch 1/20
219/219 [==============================] - 13s 47ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6697 - BBOX_Head_accuracy: 0.7717 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 2/20
219/219 [==============================] - 9s 41ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 3/20
219/219 [==============================] - 9s 40ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 4/20
219/219 [==============================] - 8s 37ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 5/20
219/219 [==============================] - 9s 42ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 6/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 7/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 8/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 9/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 10/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 11/20
219/219 [==============================] - 9s 43ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 12/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 13/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 14/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 15/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 16/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 17/20
219/219 [==============================] - 9s 43ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 18/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 19/20
219/219 [==============================] - 8s 38ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067
Epoch 20/20
219/219 [==============================] - 8s 36ms/step - loss: nan - Classification_Head_loss: nan - BBOX_Head_loss: nan - Classification_Head_accuracy: 0.6706 - BBOX_Head_accuracy: 0.8014 - val_loss: nan - val_Classification_Head_loss: nan - val_BBOX_Head_loss: nan - val_Classification_Head_accuracy: 0.6807 - val_BBOX_Head_accuracy: 0.8067

# plot the loss
import matplotlib.pyplot as plt
plt.plot(History.history['Classification_Head_loss'], label='train Classification_Head_loss')
plt.plot(History.history['val_Classification_Head_loss'], label='val_Classification_Head_loss')
plt.plot(History.history['BBOX_Head_loss'], label='train BBOX_Head_loss')
plt.plot(History.history['val_BBOX_Head_loss'], label='val BBOX_Head_loss')

plt.legend()
plt.show()
plt.savefig('LossVal_loss')

# plot the accuracy
plt.plot(History.history['Classification_Head_accuracy'], label='train Classification_Head_accuracy')
plt.plot(History.history['val_Classification_Head_accuracy'], label='val_Classification_Head_accuracy')
plt.plot(History.history['BBOX_Head_accuracy'], label='train BBOX_Head_accuracy')
plt.plot(History.history['val_BBOX_Head_accuracy'], label='val_BBOX_Head_accuracy')
plt.legend()
plt.show()
plt.savefig('AccVal_acc')

<Figure size 720x288 with 0 Axes>

Model testing

y_pred= model.predict(testImages[testLabels])

y_pred[0]

array([[nan],
       [nan],
       [nan],
       ...,
       [nan],
       [nan],
       [nan]], dtype=float32)

y_pred[1]

array([[nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       ...,
       [nan, nan, nan, nan],
       [nan, nan, nan, nan],
       [nan, nan, nan, nan]], dtype=float32)

History.history

{'BBOX_Head_accuracy': [0.7717142701148987,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607,
  0.8014285564422607],
 'BBOX_Head_loss': [nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan],
 'Classification_Head_accuracy': [0.6697142720222473,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622,
  0.6705714464187622],
 'Classification_Head_loss': [nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan],
 'loss': [nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan],
 'val_BBOX_Head_accuracy': [0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668,
  0.8066666722297668],
 'val_BBOX_Head_loss': [nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan],
 'val_Classification_Head_accuracy': [0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701,
  0.6806666851043701],
 'val_Classification_Head_loss': [nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan],
 'val_loss': [nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan,
  nan]}

END